On unlearnable problems -or- A model for premature saturation in backpropagation learning

In this paper we investigate the learning of an unlearnable problem and how this relates to the premature saturation of hidden neu-rons in error backpropagation learning. General aspects of our model are discussed. A sketch of the derivation of equations for the development of the signiicant weights in time is given. 1. Introduction The phenomenon of premature saturation of hidden neurons in feedforward neural networks trained by error backpropagation learning has repeatedly been reported by diierent researchers 2]. Diierent approaches have been proposed to circumvent this severe problem that can prevent proper learning. In 4] it is stated that the saturation is due to improperly chosen initial weights, where improper is to be regarded with respect to network parameters. We show that the relationship between these network parameters and the data to be learned is the major eeect leading to the undesirable growth of some weights. Therefore we will suggest and discuss a model for an extremly diicult learning task, relate it to backpropagation learning and then sketch the derivations of equations for the weight development during saturation. 2. The Model During many experiments reported elsewhere 1], we could observe that the probability for premature saturation depends on the relationship between the network parameters and the data, with which the network is to be trained. Especially when a problem is diicult to learn for a network (which does not imply that the chosen connguration is not well suited to accomplish the task), saturation can be observed. An extreme task that can never be learned by any This work has partially been funded by the German Federal Ministry education science research and technology as part of the AENEAS project, grand number 01 IN 505 C 4.